基于多智能体强化学习的配电网电压分散控制

doi:10.3969/j.issn.2097-0706.2024.10.005

综合智慧能源 ›› 2024, Vol. 46 ›› Issue (10): 32-39.doi: 10.3969/j.issn.2097-0706.2024.10.005

基于多智能体强化学习的配电网电压分散控制

马刚¹^,²(), 马健², 颜云松¹, 陈永华¹, 赖业宁¹, 李祝昆¹, 唐靖¹

1.国电南瑞科技股份有限公司电网运行风险防御技术与装备全国重点实验室，南京 211106
2.南京师范大学电气与自动化工程学院，南京 210023

收稿日期:2024-07-15 修回日期:2024-08-27 接受日期:2024-09-13 出版日期:2024-10-25
作者简介:马刚（1984），男，教授，博士，从事新能源发电及并网研究，nnumg@njnu.edu.cn；
马健（2001），男，硕士生，从事电力系统安全稳定控制研究；
颜云松（1981），男，正高级工程师，硕士，从事电力系统安全稳定控制研究；
陈永华（1979），男，正高级工程师，从事电力系统安全稳定控制研究；
赖业宁（1975），男，正高级工程师，博士，从事电力系统安全稳定控制研究。
基金资助:
江苏省重点研发计划项目(BK20232026);智能电网保护和运行控制国家重点实验室课题(SGNR0000KTTS2302147)

Decentralized voltage control of distribution network based on multi-agent reinforcement learning

MA Gang¹^,²(), MA Jian², YAN Yunsong¹, CHEN Yonghua¹, LAI Yening¹, LI Zhukun¹, TANG Jing¹

1. State Key Laboratory of Technology and Equipment for Defense Against Power System Operational Risks， Nari Technology Company Limited， Nanjing 211106， China
2. School of Electrical and Automation Engineering，Nanjing Normal University， Nanjing 210023， China

Received:2024-07-15 Revised:2024-08-27 Accepted:2024-09-13 Published:2024-10-25
Supported by:
Science and Technology Project of Jiangsu Province(BK20232026);State Key Laboratory of Smart Grid Protection and Operation Control(SGNR0000KTTS2302147)

摘要/Abstract

摘要：

大规模分散资源接入配电网改变了传统配电网的潮流分布，导致电压频繁越限。以模型为基础的电压控制方法对电力系统网络拓扑结构要求较高，求解时间较长，不能达到电压实时控制要求。为此，提出一种考虑异步训练的多智能体在线学习配电网电压分散控制策略。该方法将每个光伏逆变器都视为一个智能体。首先对智能体进行分区调整，然后将配电网的电压无功控制问题建模为马尔可夫决策过程，在满足系统分布式约束的基础上，采用多智能体强化学习分散控制框架，结合多智能体深度确定性策略梯度算法对多智能体进行训练。经过训练的智能体可以不需要实时通信，利用局部信息实现分散决策，制定光伏逆变器的出力计划，做到电压实时控制，减少网络损耗。最后，通过仿真验证了该方法的有效性和鲁棒性。

关键词: 配电网, 多智能体, 电压分散控制, 多智能体深度确定性策略梯度算法, 马尔可夫决策过程

Abstract:

The integration of large-scale decentralized resources into the distribution network has changed the traditional power flow distribution， resulting in frequent voltage violations. Model-based voltage control methods require a detailed knowledge of power system network topology and have long computation time， making them unsuitable for real-time voltage control.To address this， this paper proposes a multi-agent online learning strategy for decentralized voltage control in distribution networks， considering asynchronous training. The method considered each photovoltaic （PV） inverter as an agent. First， the agents were partitioned and adjusted， then the voltage reactive power control problem of distribution network was modelled as a Markov decision process. Based on distributed system constraints， a multi-agent reinforcement learning decentralized control framework was used， and agents were trained with a multi-agent deep deterministic policy gradient（MADDPG） algorithm. Once trained， the agents could make decentralized decisions using local information without real-time communication， enabling real-time voltage control and reducing network losses by determining the output plan for the PV inverters. Finally， the effectiveness and robustness of the method were verified through simulation.

Key words: distribution network, multi-agent, voltage decentralized control, multi-agent deep deterministic policy gradient, Markov decision process

中图分类号:

TM714

马刚, 马健, 颜云松, 陈永华, 赖业宁, 李祝昆, 唐靖. 基于多智能体强化学习的配电网电压分散控制[J]. 综合智慧能源, 2024, 46(10): 32-39.

MA Gang, MA Jian, YAN Yunsong, CHEN Yonghua, LAI Yening, LI Zhukun, TANG Jing. Decentralized voltage control of distribution network based on multi-agent reinforcement learning[J]. Integrated Intelligent Energy, 2024, 46(10): 32-39.

图/表 11

图 1

图2

图3

图4

图5

图6

表1

图7

图8

图 9

图10

参考文献 20

[1]	黄维维, 张安安, 敬佳佳, 等. 微电网电压/无功协调控制综述[J]. 电网与清洁能源, 2016, 32(3):50-57.
	HUANG Weiwei, ZHANG An'an, JING Jiajia, et al. An overview on micro‑grid voltage/reactive power coordinated control[J]. Power System and Clean Energy, 2016, 32(3):50-57.
[2]	王林, 孔小民, 周忠玉, 等. 云储能模式下的配电网分布式光伏-储能无功优化方法[J]. 综合智慧能源, 2024, 46(6):44-53. doi: 10.3969/j.issn.2097-0706.2024.06.006
	WANG Lin, KONG Xiaomin, ZHOU Zhongyu, et al, Distributed photovoltaic-energy storage reactive power optimization method for distribution networks under cloud energy storage mode[J]. Integrated Intelligent Energy, 2024, 46(6):44-53. doi: 10.3969/j.issn.2097-0706.2024.06.006
[3]	林泓宏, 余涛, 张桂源, 等. 基于数据驱动的高比例新能源配电网无功优化算法[J]. 综合智慧能源, 2023, 45(11):10-19. doi: 10.3969/j.issn.2097-0706.2023.11.002
	LIN Honghong, YU Tao, ZHANG Guiyuan, et al. Data‑driven reactive power optimization algorithm for the distribution network with high proportion of renewable energy[J]. Integrated Intelligent Energy, 2023, 45(11):10-19. doi: 10.3969/j.issn.2097-0706.2023.11.002
[4]	李鹏, 姜磊, 王加浩, 等. 基于深度强化学习的新能源配电网双时间尺度无功电压优化[J]. 中国电机工程学报, 2023, 43(16):6255-6266.
	LI Peng, JIANG Lei, WANG Jiahao, et al. Optimization of dual‑time scale reactive voltage for distribution network with renewable energy based on deep reinforcement learning[J]. Proceedings of the CSEE, 2023, 43(16):6255-6266.
[5]	郑许林, 袁源, 柏晶晶, 等. 分布式光伏无功电压主动支撑控制方法[J]. 可再生能源, 2023, 41(12):1685-1690.
	ZHENG Xulin, YUAN Yuan, BAI Jingjing, et al. Active support control method of distributed photovoltaic reactive power and voltage[J]. Renewable Energy Resources, 2023, 41(12):1685-1690.
[6]	潘思蓉, 刘友波, 唐志远, 等. 含深度学习代理模型的有源配电网电压无功控制进化算法[J]. 电力系统保护与控制, 2022, 50(17):97-106.
	PAN Sirong, LIU Youbo, TANG Zhiyuan, et al. An evolutionary algorithm for Volt/Var control in an active distribution network with a deep learning surrogate mode[J]. Power System Protection and Control, 2022, 50(17):97-106.
[7]	DUAN J J, SHI D, DIAO R S, et al. deep‑reinforcement‑ learning‑based autonomous voltage control for power grid operations[J]. IEEE Transactions on Power Systems, 2020, 35(1):814-817.
[8]	邵美阳, 吴俊勇, 石琛, 等. 基于数据驱动和深度置信网络的配电网无功优化[J]. 电网技术, 2019, 43(6):1874-1885.
	SHAO Meiyang, WU Junyong, SHI Chen, et al. Reactive power optimization of distribution network based on data driven and deep belief network[J]. Power System Technology, 2019, 43(6):1874-1885.
[9]	ZHANG Y, WAN X N, WANG J H, et al. Deep reinforcement learning based Volt-VAR optimization in smart distribution systems[J]. IEEE Transactions on Smart Grid, 2021. 12(1): 361-371.
[10]	史景坚, 周文涛, 张宁, 等. 含储能系统的配电网电压调节深度强化学习算法[J]. 电力建设, 2020, 41(3):71-78. doi: 10.3969/j.issn.1000－7229.2020.03.009
	SHI Jingjian, ZHOU Wentao, ZHANG Ning, et al. Deep reinforcement learning algorithm of voltage regulation in distribution network with energy storage system[J]. Electric Power Construction, 2020, 41(3):71-78. doi: 10.3969/j.issn.1000－7229.2020.03.009
[11]	FERREIRA L R, AOKI A R, LAMBERT-TORRES G. A reinforcement learning approach to solve service restoration and load management simultaneously for distribution networks[J]. IEEE Access, 2019,7:145978-145987.
[12]	张志荣, 邱晓燕, 孙旭, 等. 基于ADMM的交直流混合配电网分布式电压优化模型[J]. 电网技术, 2021, 45(11):4551-4559.
	ZHANG Zhirong, QIU Xiaoyan, SUN Xu, et al. Distributed voltage optimization model for AC/DC Hybrid distribution network based on ADMM[J]. Power System Technology, 2021, 45(11):4551-4559.
[13]	邓清唐, 胡丹尔, 蔡田田, 等. 基于多智能体深度强化学习的配电网无功优化策略[J]. 电工电能新技术, 2022, 41(2):10-20. doi: 10.12067/ATEEE2103019
	DENG Qingtang, HU Dan'er, CAI Tiantian, et al. Reactive power optimization strategy of distribution network based on multi agent deep reinforcement learning[J]. Advanced Technology of Electrical Engineering and Energy, 2022, 41(2):10-20. doi: 10.12067/ATEEE2103019
[14]	张沛, 朱驻军, 谢桦. 基于深度强化学习近端策略优化的电网无功优化方法[J]. 电网技术, 2023, 47(2):562-572.
	ZHANG Pei, ZHU Zhujun, XIE Hua. Reactive power optimization based on proximal policy optimization of deep reinforcement learning[J]. Power System Technology, 2023, 47(2):562-572.
[15]	LIU H T, WU W C. Online multi‑agent reinforcement learning for decentralized inverter‑based Volt-VAR control[J]. IEEE Transactions on Smart Grid, 2021, 12(4):2980-2990.
[16]	刘硕, 冯斌, 郭创新, 等. 考虑自注意力和时序记忆的深度强化学习配电网无功电压控制策略[J/OL]. 中国电机工程学报,(2023-10-09)[2024-05-06]. http://kns.cnki.net/kcms/detail/11.2107.tm.20231007.1822.004.html.
	LIU Shuo, FENG Bin, GUO Chuangxin, et al. Volt-Var control strategy of distribution network based on deep reinforcement learning considering self‑attention and temporal‑memory[J/OL]. Proceedings of the CSEE,(2023-10-09)[2024-05-06]. http://kns.cnki.net/kcms/detail/11.2107.tm.20231007.1822.004.html.
[17]	KABIRI R, HOLMES D G, MCGRATH B P, et al. LV grid voltage regulation using transformer electronic tap changing, with PV inverter reactive power injection[J]. IEEE Journal of Emerging and Selected Topics in Power Electronics, 2015, 3(4):1182-1192.
[18]	王康平, 张兴科, 刘财华, 等. 基于自适应下垂控制的风电场无功电压控制策略[J]. 综合智慧能源, 2022, 44(4):12-19. doi: 10.3969/j.issn.2097-0706.2022.04.002
	WANG Kangping, ZHANG Xingke, LIU Caihua, et al. Reactive power and voltage control strategy based on adaptive droop control for wind power plants[J]. Integrated Intelligent Energy, 2022, 44(4):12-19. doi: 10.3969/j.issn.2097-0706.2022.04.002
[19]	郑晓莹, 陈政轩, 曾琮. 含分布式电源的配电网双层分区调压策略[J]. 电力系统保护与控制, 2021, 49(6):90-97.
	ZHENG Xiaoying, CHEN Zhengxuan, ZENG Cong. Double‑layer partition voltage regulation strategy of a distribution network with distributed generation[J]. Power System Protection and Control, 2021, 49(6):90-97.
[20]	刘朝阳, 穆朝絮, 孙长银. 深度强化学习算法与应用研究现状综述[J]. 智能科学与技术学报, 2020, 2(4):314-326. doi: 10.11959/j.issn.2096-6652.202034
	LIU Zhaoyang, MU Chaoxu, SUN Changyin. An overview on algorithms and applications of deep reinforcement learning[J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(4):314-326.

控制方法	平均电压偏差（标幺值）	电压可控率/%	总网损/MW	平均时间/ms
无控制	0.639 4	72.35	0.128 6
MADDPG	0.302 2	89.67	0.093 7	2.47
IPPO	0.384 5	80.74	0.149 1	7.36
OLDC-MADDPG	0.285 2	90.43	0.067 8	2.83

基于多智能体强化学习的配电网电压分散控制

Decentralized voltage control of distribution network based on multi-agent reinforcement learning

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 20

相关文章 15

编辑推荐

Metrics

本文评价

[1]	王林, 孔小民, 周忠玉, 刘建平, 王晓东, 张宁. 云储能模式下的配电网分布式光伏-储能无功优化方法[J]. 综合智慧能源, 2024, 46(6): 44-53.
[2]	冯骥, 杨国华, 史磊, 潘欢, 陆宇翔, 张元曦, 李祯. 基于并行融合深度残差收缩网络的有源配电网故障诊断[J]. 综合智慧能源, 2024, 46(6): 8-15.
[3]	朱卫卫, 朱清, 高文森, 刘财华, 王录泽, 刘增稷. 基于5G通信时延的配电网馈线自动化切换方法[J]. 综合智慧能源, 2024, 46(5): 1-11.
[4]	杜龙, 沙建秀, 樊贝, 胡静威, 刘增稷. 基于信息物理双侧数据的配电网CPS窃电检测方法[J]. 综合智慧能源, 2024, 46(5): 20-29.
[5]	李一诺, 刘玮, 魏兴慎, 王琦. 网络攻击场景下含分布式光伏配电网的脆弱性研究[J]. 综合智慧能源, 2024, 46(5): 50-57.
[6]	陆文甜. 基于增量交换的主动配电网分布式多目标最优潮流[J]. 综合智慧能源, 2024, 46(2): 43-48.
[7]	郁海彬, 董烨, 翁锦德, 胡忻晨, 严威, 吴迪凡. 电力5G切片在城市配电网中的应用及经济效益研究[J]. 综合智慧能源, 2024, 46(1): 75-83.
[8]	刘艺娴, 王玉彬, 杨强. 基于门控图神经网络的高容错配电网状态估计方法[J]. 综合智慧能源, 2023, 45(6): 1-8.
[9]	刘子祺, 苏婷婷, 何佳阳, 王裕. 基于多目标粒子群算法的配电网储能优化配置研究[J]. 综合智慧能源, 2023, 45(6): 9-16.
[10]	林泓宏, 余涛, 张桂源, 张孝顺. 基于数据驱动的高比例新能源配电网无功优化算法[J]. 综合智慧能源, 2023, 45(11): 10-19.
[11]	吴雪琼, 夏栋. 新型主动配电网智能规划与决策技术研究综述[J]. 综合智慧能源, 2023, 45(11): 20-26.
[12]	郑真, 朱峰, 马小丽, 田书欣, 姜皓喆. 基于TL-LSTM的新能源功率短期预测[J]. 综合智慧能源, 2023, 45(1): 41-48.
[13]	胡祖源, 靳现林, 谭雅之, 樊静宜. 基于改进粒子群算法的分布式光伏及储能系统优化配置[J]. 综合智慧能源, 2023, 45(1): 49-57.
[14]	华咏竹, 谢强强, 秦会斌, 邵李焕, 崔佳冬. 计及用户端调节容量的变频空调自适应电压调控策略[J]. 综合智慧能源, 2022, 44(2): 21-28.
[15]	王卓容, 石庆鑫. 基于拓扑重构与分布式电源调度的配电网灾后恢复策略[J]. 综合智慧能源, 2022, 44(2): 29-34.